This invention relates generally to telecommunications and specifically to a method and system for evaluating the quality of packet-switched voice signals.
Telephone connections have always been subject to impairments in the form of noise, attenuation, distortion, crosstalk, and echo. Such impairments are particularly common to analog portions of the connection, such as along subscriber loops and within frequency domain multiplexing equipment. Digital transmission alleviates many of these problems but introduces quantization noise or glitches. Even with perfect digital transmission applied for long-haul connections, a typical telephone connection includes many analog components, such as hybrids, where impairments can occur.
A poor connection or a malfunctioning piece of equipment can produce conditions that a telephone customer will find objectionable or intolerable, so that the connection is deemed to be of very poor quality. When there is a high incidence of such poor connections, customers may complain to the service provider or regulatory authorities, or simply change long distance carriers. Perceived quality of telephone connections is therefore a major factor affecting the reputation and marketability of long distance telephone services.
To guard against poor quality, telephone service providers have developed methods to take objective quality measurements upon a line, a piece of equipment, or even an end-to-end telephone connection. These measurements can help the service provider detect and gauge impairments, pinpoint weak elements and correct deficiencies that degrade user perception of quality.
Many such objective measurements are well known and standardized in the art. Empirically-derived thresholds enable analysts to infer the existence and severity of quality problems by comparing measurements to tables of acceptable value. For example, power levels of test signals and quiet channel noise can be measured electronically. Since it is well known that a certain range of signal levels must reach a telephone receiver to produce acceptable volume at the earpiece, and C-weighted noise must be kept at a level relative to the signal level to keep users from experiencing unacceptable noise at the earpiece, the combination of measures for a particular connection supports accurate assessment of the likelihood that a user would find that connection to be of poor quality with respect to ability to hear the distant talker.
For such objective measurements, the effect of extreme values on user perception of quality is clear, and there are easily discerned thresholds for xe2x80x9cno effectxe2x80x9d and xe2x80x9csubstantial degradationxe2x80x9d conditions. However, for immediate values, there is generally no clear division between values representing acceptable and unacceptable connection quality. Speech clarity and perceived connection quality depend on many variables, including, for example, speech content, talker rhythms, subjective perception of the listener, and users"" acclimation to their telephone service. As a result, the correlations between values of objective measures and user perception of connection quality are statistical, representing the combined effects of many different kinds of impairments and variations of sensitivity to them among the population.
Earlier work by the inventor and others in this area have created mappings between objective measurements and perceived quality, so that, for example, when a given circuit was measured in terms of signal level, noise, distortion, crosstalk, and echo, the mapping predicts the percentage of conversations that would be reported as being significantly impaired or of poor quality as perceived by an average user population. Such mappings have proven to be a powerful tool for analyzing reported impairments and for gauging acceptable performance of a new line or piece of equipment before deployment.
The mapping was produced by creating or finding various combinations of measurable characteristics along telephone circuits and then having a population of callers conduct test calls to subjectively gauge the quality of each call. For each test call, the circuit under analysis was rated on a scale of None-Some-Much for each of the impairments manifested to users of the connection related to the selected objectively measurable characteristics. These impairments include noise, volume, distortion, and echo. Each caller also provided an opinion score, which is an overall rating of the circuit quality on a numerical scale.
Each caller also determined whether the overall effect of the impairments was to render the connection: unusable (U; rendering the channel entirely unusable), difficult (D; causing enough difficulty to require adaptation by the speaker and listener), irritating (I; disturbing but not requiring adaptation by the speaker and listener), noticeable (N; minor enough to be ignored), or unnoticeable (O; no effect on quality). The percentage of calls or connections that elicit any one of the first three responses (unusable, difficult, or irritating) is called the P(UDI). The P(UDI) is of particular interest to service providers as a meter of customer satisfaction because it has been shown that overall satisfaction decreases as P(UDI) increases, regardless of average opinion score.
Analysis of empirical data including user reports of impairments and perception quality, together with user reports of impairments obtained in conjunction with objective measurements of connection characteristics then supported a two step development of a means for predicting user perception of quality from objective measurements. First, a model supporting prediction of P(UDI) and average opinion score as a function of percentages of calls with each of the possible combinations of xe2x80x9cnone,xe2x80x9d xe2x80x9csome,xe2x80x9d and xe2x80x9cmuchxe2x80x9d (N, S, M) conditions reported for each of the impairments considered was produced. Then, objective measurements were correlated with user reports of impairments to predict the proportion of N, S, M reports from users that will eventuate as a function of objective measurements. From these two elements, it was then possible to take measurements of the objective characteristics for connections and translate the set of measures obtained into estimates of likely user perception of quality as revealed by P(UDI) and the average opinion score.
In one aspect, the present invention provides a technique for assessing the quality of a packet-switched communications channel. For instance, the technique provides a means by which a set of objective measurements for a packet-switched telephony connection can be processed to derive a projected perceptual quality level for the connection. It extends upon the techniques of the prior art to adapt for the effects of important new technologies.
Specifically, the present invention addresses packet-switched (e.g., Internet Protocol based) telephony, which is subject to a different set of impairments from ordinary link-switched telephony. The packet-switched environment can produce momentary interruptions or latencies in a streaming signal. In addition, some speech compression and coding schemes can suffer dropouts or garbled portions depending upon the loss of certain packets. Such impairments represent new phenomena that must be incorporated into the measures-to-impairments and impairments-to-quality transforms used to predict user perception of quality.
In one embodiment, the present invention provides a method of evaluating the quality of a packet-switched voice signal. To begin, a plurality of objective characteristics for a voice signal being transmitted across a packet-switched network are selected. At least one of these objective characteristics is packet loss. For each of the objective characteristics, correlations are determined in order to categorize quantitative measurements of the objective characteristic into a number of categories, representing a subjective assessment of the presence and severity of impairments as perceived and described by users. To do so, a plurality of evaluation voice messages is generated by varying selected ones of the plurality of objective characteristics for each of the evaluation voice messages. The quality of each of the evaluation voice messages is then empirically measured by having a test subject (or subjects) listen to the evaluation voice messages and grade each voice message with respect to the selected impairments. The probability that a packet-switched voice signal will achieve a particular rating is then determined, by suitably amending extant effects models, which correlate subjective assessments of presence and severity of impairments with subjective measures of overall connection quality. The determination thus utilizes a measurement of objectively quantifiable characteristics of the packet-switched voice signal to predict the subjective user quality rating of the packet-switched voice signal.
In one embodiment, the objective characteristics are correlated with corresponding user perception of the severity of the manifestation of that characteristic as an impairment to conversation. For example, if a packetized voice signal is missing some packets, a listener may perceive speech distortion. In this case, the objective characteristic is packet loss and the manifestation of that characteristic to the user is speech distortion.
The techniques of the present invention allow objective measurements to replace subjective measurements upon a channel. This yields an improvement, for example, in gauging performance of a channel or a communications element before it is put into service. The objective assessment of an Internet telephony channel is expressed in a way that is relevant to perceived quality.
The present invention can be used in a variety of contexts. For example, a phone system or a portion of a phone system (e.g., a line or a piece of equipment) can be tested before being used commercially. In another application, a workstation stores and analyzes objective measurements. This workstation can be used by an engineer troubleshooting a network or a sales engineer comparing qualities of various options. The present invention could also be used in an automatic, real-time control system for a communications network.